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ABSTBICT 

' This papet iSescribes a garadig* for tutorial systems 
capable of aatosat ically providing feedback and hints in a game 
environment. Thg paradigi is illustrated by a tijtorin^ system for the 
PIATO gam< *«Hov the 8est Was Bon." The system uses a computer-based 
"Expert** player to evaluate- a s^adent*s moves «nd construct a 
".diffe'rential mo^l" of the stroent's behavior ifith respect to the 
Expert's. The essential aspects of the student's behavior are " 
•analymed with respect to a set of "issues," vhich are addressed to 
the basic conceptual ,constraints/that might prevent the student's 
fall utilization of the enviroiyient. Ijssuejs are vieved as procedural 
•specialists that "wake-up" or become active ifhen an instance of an 
l^ssue manifests itself in a move. These issue i^pecialists help the 
Ttitor isolate what to comment on. The intent of the system is to , 
transfo'rf a -"fun" game into a productive learning environment without 
altering the student's enjoyment.^ (Author) 
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> »Abst!jact 

This pappr describe? a paradigm for' tutorial systems'' capabler, of\ 
automatically providing feedback and hints in a gane en virontnent • The paradigm 
is illustrated by a tutoring systel?) for tlje PLATO gam^ "How the West- Was Won". 
The system uses a coi1:puter-based "Expert" player to evaluate a student's moves 
and construct a "differential model" of the student's behavior with reaped to 
the Expert's. The essentiai aopects of th'e student's behavior are analyzed with 
respect to a s^t of '^issues.", which are- addressed to ' the basic conceptual 
constraints -that night prevent the stucj^nt 's full -utilization of* the 
environment.^ Issues are viewed as procedural specialists that "wake-up" or 
become activis when an instance of an issue manifests itself in a move. Theae 
issue speciaJLists help the Tutor i^i'olate what to comment on.^ The intent of the 
^system is to transform a "fun" game into a product ive 'learning; environment 
without altering the student's enjoym'ent. ^ ^ v , • 
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use of such "unstructured gaming 
environments" is tiie amount of- teacher 
attention that is often^-^equired • tp keep 
the' student from ^ormin^^jST ossly incorrect 
models of the underlying struct ure of the 
gaAe and to •identify interesting 
shortcoinin^s of particular strategies. In 
brief, fpr^a gaming ^environment tb be fully 
utilizec* as a learning instruneht the 
environment must be augmented by tutorial 
gui<^ance which points out weaknesses in the- 
studont 's ideas or suggests ideas when the 
student appears to have none\ This paper 
presentg^^ a paradigm for designing computer 
systems capable of providing this -kind of 
tutorial guidance and describes an example 
of one such system built around a drill and 
practice game* in arithmetic. 

J3ftfore describing our ba^ic paradigm 
for construci^g t^utoring systems, we want 
to stress the difiTerence between the notion 
of tutorial behavior as used' here and that 
which has previously been proposed- In 
classica»l \CAI , the tutoring bejia'vior is 
"locally controlled ' by predetermine i 
brahching point^s in an^ Inst ruct ;Lonal 
sequence. The inst ruotTonal sequence is 
restricted to the extent that' each 
branching point is testing ^ for the 
understanding * of a small number -of 
concepts. The author of the s%quence is 
then abl.e to predict which nisconceptions 
enticing problem solving eri^ironmcnt Which lead to what response.s and" branch to ^'the 
the student 'can explore aT'will, creating propA' remedial sequences.- In ^the^ gaming 
his own ideas of its underlying structure environment, ttie course pf the" game is. 
and dynthesizing stratep:ies which reflect determined^ largely 'by the student. The 
his understanding of tnir. structure. Games' tutoring module is given •freedom to 
also have t^e ^pot<5lntial -for mo^Livating" interrupt .the student at any tfrrve and make 
drill' and .-practice by providing suggestions ipr correct misconceptions, but 
environments in whicta students acLuc^lly it cannot take control, of the game away 
e'njoy repetition. However, in bath cases, ' * *- 

a major stumbling, block to ,^h^ efj^octive 
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JODU^CTION 



An exciting and relatively unexplored 
use of computers in edUcati)n\ involves 
coupling an^aSaptive tutor (-or comr.onta«tor ) 
to an educational* gan€ . Games provide an 
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from the student. That is^^ the tutoc*ing 
^module has, no specified branching , points or 
* any other explicit (as opposed to implicit) 
means ,,for direct'in^ the game - into, 
particular situations. Hence a major* 
challenp,e in creating this kind of tutor is 
to enable iU to use' its knowled^^e of the 
'doraain* together with a synthesized model of 
• the student's pa&t behavior to decide what 
to say and when to say I't . The' tutor must 
be percept^ive- enough to make relevaftt 
comments but at the same time it must not 
be so intrusive ^s ^to destroy ^he fun 
inherent in the game.* * - 



. The viability o'f this appj^ch -deoends 
critically on techniques for JPltonatically 
inducing a ''model*' of the ^ student ' which 
accurately represents his reasoning 
strategies and current ftate of knowledge. 
If tjie computer-based tutor is to deviate 
from (or not to use at all) a predetermined 
*inst,ruct ional- sequence, its new course of 
action must be based not only on its 
reasoning c? oabilit t^s' but also on the 
details, of a student 's observed strengths 
and weaknesses and ^ny shortcomings 
manifested in his current "move." 



Tutoring by Issue and Example z±. a. 
Paradigm 



General 



The paradigiB of "issues and .examples" 
developed to focus the tutoring system 



^as 
on relevant 
behavior. Th 
concepts) of 'the domain 
student is expected to 



poMions of 
1^ important a 



the 
D.e c t s 
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know 
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or 



student s 
(skills or 
what the^ 
learn) are 
of "issues" . 
pacts 6/ the 
are monitored by th^ 
is aC't^ivated by patterns 
student s behavior ^^or 
student use^ or does not/ 



identified as a collection 
The^^^jLSsucs determine what 
student '^s behavior 
tutjor. Each is3ue 
which watch *tne 
evidence that the 
use their particular concept or skill. As 
the student pl-ays, a model of how he is 
performing, 'With respect* to each issue,* is 
coYist ^ucted^ When h^. makes a /'bad" move, a 
tutorial prorr^r;* uses the model to ^ .decide 
why the student did not make a better^'jnove, 
that is, whiph isnuehe missed. Once an 
fssue has ,^bceYi^ dete^f^ined, the tutor can 
prJpsent an explanation of that iss^e 
together • with a better move ' whicn 
illustrates -the issue. In this way,^ the 
student' can see 'the usefulness of - the 
"issue" at a time^ when he- wi 1 be most' 
receptive to the idea presented 
immediately after he-has though^ -about the 
problem. * 

Figure ' 1 is a diagram of Ih^ 
modelling/tutorial ^ process q^derlying the 
paradigm. Figure la presents the process 
of constructing of a model of tne student's 
behavior., The mod^fi^ is a summary of - the 
student's per'formance whj,le solving^ a 
ri'es of problems (in this case, moves ip 
a Nzame), Each - time the student makes. a 




'm(yre, he exhibits a c/rtain behavior., The 
important asp'ects of this behavior (the'' 
issues) are abstracted by the pattern 
matching conponent of each issue (called 

•the "recognizer"), s This 'abstracting is 
also done with respect to th^ behavior of a 
computer-based "Experit" in the sa;ne 
environment by the sane - recognizers. The 
two abstractions are compared to provide a 
di f f er cnt lal modeT of the studeat 's 
behavior , ' which i^ndLcates those issues on 
whi,,ch the student is weak. Notice that 
without the Expert it is not possible to ^ 
determine whether the student' is weak in 
some area or whether, the need for that 
skill has arisen InfreguentLy in the 
student ^s experiene^. 



Figure 1, Diagram of the Modelling/ 
Tutoriel Process 
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Figvre lb presents the tutoring 
processT ' When * the student makers a l^ess 
than ^optimal • move (a3 determj'ned by 
-comparing his ^ mpve with those of the 
Expert), the 'tutor - uses the model 
evaluation component of each issue (called 
the "evaluator") to scan the student mode^ 
^nd 'to create^a list of issues on whic+i the. 
^tudent- is weak. From the Expert's list of 
* better maves, the tutor uses the "issue" 
^ recognizers t'p determine which issues are 
illustrated by better moves.- From these 
two lists ( the- f'weak" • issues ^nd'the better 
move issues'),, .the tut^/ select.s an issue 
and va good move which illustrates it) The • 
selected issue and example arc then passed 
to the outpiit generators which produce the 
feec^bafbk to^ the student. 

/ We would like to stress two point^ in 

tHe above* process . 'One*is the necessity of 
the Expert and the other is- the importance 
of l,d^tifying the crit ical. issues . ^ ,The 
Expert provides a ^^qasur* for eva'luatmg 
the #5tiident»'s behavior in novel situation^ 
without 'wiUc+i it would be necessary tc^ 
severely restrict thef game situations whic^ 
. could be . tutor*^^,^ The issues define tltpse 
'conceptual co^npotfe^s of th^ environnent 
ji^hich the student is expected to leacn and 
* the^r provide the tutor a handle/ to 
structure and direct the exploration of the " 
environment by thie student . 

II, AN EXAMPLE SYSTEM 
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In order to explore the ramifications 



effectiveness t)f 



the 



issue 



anc 



examplcA" paradign, we chose a domain in 
^ which we could easily construct an expert 

program that the tutor oould call . on for- 

evaluating the student's behavior. The 
- domain of knowledge chosen was t-he PLATO 

gkme "How'fehe V/est V/as Won."* • ' 

- Dcs'bription of "How th# Weij't Wc^*g Won" 

* "How the West Was Won" >^ (herea/tor 
called West) Is a ^ane for two players' (the' 
computer;; usually being one ) • ' It is played 
oo a game board like that in Figure 2.'- The 
object of the'game is to get to the -last 
town* on the map (position 70). On each 
turr)'^ player gets t^hree 'spinners (rando^j 
numbers). He.'C^n combine the valyes of the 
spinj^ers using iny two (dif fei'^ent ) 
arithmetic operators ( + , or V)/ The 

' value of Ihc arithnet;c expression T^e'fma-kes 
Is" the rfunber spaces ^he gets to\novo. 

(lie must also say what the dnswer is.) If 
he makes a ^negative nunber, he m5Ves 
backwards. 

AlOn'fe the way^ there are ^shortcuts / arfd 
towns. ^If a player lands on a'^j^hortcut, he. 
' - ^ \ 

•Thi'3 game-^ras written by Bonnie Anderson' 
for-', the PLATD flemcntary Mathematics 
Project'. * '• • ■ \ 
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advances .M^o the other end (e.g. from 5 to 
13 in Figure 2). If -he Ifinds on a town^ he 
goes on to the^next town..' When a*' player 
lands on the same place as his Qpponent^ 
unless it is a town, kfis opponent* must 
retreat back tWo towns. To win/ a player 
•jnust be the first pne to' laftct exactly on 
the last town. Both players get\the, same 
number of turns, softies are possit\le. 



Figure 2. Game board for Hqw the West ^an 
Won (from PLATO)' * 
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assuffpt 



A cerMral as3TPWption of the' Wes.tN 
tutoring system is that good tu^orini? can 
point out Structure, ih an environment ^hich 
might have otherwise been missed and so 
doing ^low the student, tcy enrich, his 
uncTerstancJing of ' (and skills in) ttie 
environment. In Plato West, 'an untutored 
(unwatched) ^tiident tends to brecome fixed 

• on a Siibs6t of the available moves .and 
hence misses the potential richness of Xhe 
jame. For example, a student may adopt the 
strategy, of adding the first two spi«ners 
and' multiplying; the result by the^ third, 
spinner, (A+i?')*C. Since the third spinner 
tends Cq be 'the largest, this .strategy is 

/close to the strategy of multiplying the , 
largest number by the sun crf^ the other .two 
numbers (which prpduces the largest 
possible- number). 'If this strategy is 
augmented by a rule' that prevents noving 
off the boara (a. simple end game strategy) 
it ^generates a respectable game. 'Notice, 
however, th,at much is missed. The .student 
is unaware of the special moves such as 
bump<s and therefore of such questions a:., 
"Is it jDetter to send n*y opponent back 14 
or get 9 ahead of him?"^In ^act, since thjrs 

. kind of student s'trategy lets him consider ' 
only One moye , he misses the whole notion * 

. of strategies for deciding between 
alternatwe' moves. Fron an arithmetic 
dr^'ill /and practice ' point 6f view, .^e is 

* performing one calculation per move instead 
of dpzens of mental calculations which he 

' would have \o per/*opm to answer 'questidns 
such as, "What numbers can I form with 
these spinners?" or "Can I make an, 8 'wi,th 
thesfe spinWers'?" By interjecting corr^n^nts 
and suggesting-^f^l^ter moves to the st'j<yent 
from time to time (thouch not too often), 
the tutor tries to widen the student's view 
of the game, hence drastically improvinp 
the drill , and practice ^aspect by gett'infeV 
him to compute many more expressions per 
move.* ^ * • 



*Mych Oft our original motivation for 
bUfilcHngT tho V/est tutoring* system stemmed 
frrpm the, doctoral thesis* of Cecily Hesnick 

^[^975] which desg-rjbes some preliminary 

.* experimen'ts quosti^ing the effectiveness, 
of t^is game as a le&rninfr or drill and 

. practice environme^n t. Rcso^ck found ^ that 
although students become inlj^nsely involved 
in 'the* game they usudllaf^ developed a 

» limited * Strategy and wouTd''-pla^ th^e pame 
Tor hour;^ using t^heir strategy regardless 
of how* 'many games they .won' or" lost. 
F«urthcrmo>e , the strategies they would lock' 
onto orten rer^ired "no' sesirch and hence 
'Only ortae arith^motic expression per mpve was 
bcipg evaU'ited. ^ ' ^ 



^Protocol ' * - . ' * • 

' 'f ^ 

^' Before ' discussing' . the * njodeljring/ 
tutoring process ia greater sietail, we 
present a protocol of an actual ^ stude;it 
playing Our* We§t .* The tutoring component 
of the system "^used for the' protocol 'was 
accelerated « to ^generate more feedback than 
normal . The n6rtnally paced tutor seldom 
hassles the student two moves in a- row. In 
'the protocol,* all lines typed by the 
student begin with "=> Annotations are 

indented . ^ . / 

WELCOME to "HOW THE V^ST WAS WON". 
WHAT- M YOUR NAME? ) 

= > BOir 

DO YOU >/AI*'T TO'-GO FI^^ST? 
= > ¥ES M . 

*IT'S YOUR TURN. . .Sf>imJERS ARE: 12 2^ 
WHAT ARITHMETIC EXPRESSION DID' YOU FORM? 
=> 1*2+2 

WHAT NUMBER DOES THAT GIVE JOU? 

^>^^ ' » 

The numbers that co4jld have been^ inS6e 
were: * ^ 

"-22-1.-3530461 

"best" move is (2*2) + 1. \. 
delta-ways: 13-1, 10-8, 6-1, 4-5, 3-2, 

2-3, 1-3 
Mov^ ranks 11 which is FAJR 

The indented information above* is 
generatdd by the EXPERT wodule on ^ach 
turn . .*This inf orma t ion is not seen by 

. the student but is used by. the TUTOR, 
MODEL EVALUATOR and ^T.UDENT flODELLER , 
.''modules. It is included here to 
indicate' the cpmpflexity of information 
that is used^^ ti^e system to follow the 
• -student 's benavior> The first line 
fives' the rangQ' qf numbers * that coult! 
legally be made usin§ the, Spinners . Tne 
•second line gives the "best" move 
" according to th6- metric of maximizing 
the difference between you and your 
opponent. \ (These twd pieces af , 
information are available to the student 
through, the "hint". key i.f he wants it.) 
The third line (beginning "delta-ways") 
, represents ' the lep^il move 
first number in each* P4i| 

* number of spaces ahead tf 

be (i.e'. « delta ) 'The second nvimber is 
the ^umbt^r „of di'fferent arithmetic 
patterns wViich could h^ve b^en used to , 
achieve tha\ lead.^ In this case* there 
is one move \hich would. "have put Bob ^3 
ahe^d Cmakinp\5 and taking the 
^ shortcut), ei^h't ^ovesJ which wpuld ha", e 
put him 10 ahead (making 0 and gettir.^ 

, advanced to the next town ) , one that 
would put him 6i^a1^ad^ etc. His move 
ahead), ranl^ed ll becai^e there were lO 
better ones. rir the rtst of , the 
protocol only ttoe Rank of tAe .student 's 
move wiii, be shpwn. 



a,ce. The 
v^s the 
stTudent would 



/ 



V. 



ERIC 



BOB is AT M TH& COMPUTER IS^AT O' ^ 

It's -Mi TURN. . SpInners arei i 2 3 

m/ EXPBESSIOtI ISu (1«2)+3 
Wttl'OH GIVES ME 5 

1 TOOK A SHOHTCUT ^' 
tHE COMPUTER IS AT 13 BOB IS AT ^ 

IT'5 YOUR .TURN,. WINNERS ARE: 2 V-^^ * 
WHAT ARITHMETIC 'EXPRESSION DID Y(/U FORM? 
=> 2«1+2 ' • ' ' . . ' ' 

yfmj: number does that (hve yOu?" 

-Bob makes another le'ss*-than«opt>irnal mpve 
but two , moves is too - early to start 
tutoring. * . . 

Move ranks 9 which is FaIr ^ 

BOB is AT 8 — THE COMPUTER IS AT 13 

SPINNERS ARE*: 1 2 'l 



IT'S MY T^RN, 
MY" EXPReISIGN IS: (U1)*2 
' WHICH GIVES 'ME 4 
THE! COflfnJJER IS AT -17 BOB IS AT 8,. 

IT'S YOIJr TURN./ SPIUMERS ^9E: 115 
WHAT ARITHMETIC EXPRESSION DID *U. FORM? 
=> 1*1+5 

WHAT NUMBER DOES THAT GIVE; YOU? . • » 

^ ' Move ranks 3 'which is GOOD 

The ISSUES are : ^ PATTERN PARENS 
STRATEGY* • • 

These are the tlutorable issues which 
"h^ve been determined by considering the 
student model . tqj^bt her with the moves 
which are* better. Bob's first thrjse 
mo^es have all had the same pattern/ . 
a*b+c which does not require 
parenthe^s; whereas the best move use3 
a difffl^ent pattern which does- require 
, fJarentheses so t^he issues 'of. PATTERN and 
PARENS are noticed. The issue of 
PATfERN .was chosen , because ^ it is mdre. 
specif\ic than PAREMS. * That is-, PATTERN 
appears ahead of* PARENS on at printing 
list* *The'^TRATEGY issue indicates that 
there is na^ coherent strat^y arising- 
for the i series of moves made by th'e 
student, as Hill be, disbursed later* 

. IT'S BEGINNING TO SEEM LIKE YOU STICK TO 
THE SAME OLD. STANDARD PATTERN IN YOUj^ MOVE. 
THE REAl^Y EXPERT PLAYERS TRY A WHOLE BUNCH 
OF POSSIBILITIES. AN EXAMPLE OF A GOOD 

•EXPRESSION WOULD BEr (1 + 1 )*5, WHICH LS A 
PRETTY GOOD HOVE. SO YOU COJiLD flAVE BEEN ^ 
AT 18 'WITH ME FINISHING UP THE TURN AT 17. 

Aftjpr the presentation of a better move, 
'the student ma/ be allowed to, retake his, 
• turn. This usually - makes thc» student 
more receptive to the advice. In this * 
case Bob decides not to put himself, \n 
front of his^ opponent. . % 



WOULD YOU LIKE TO TAKE YOUR TURN OVER? 
r> NO ' 

B'OB is at m — THE COMPUTER 13 ,AT 17 

IT'S MY ^URN.. SPI^^NERS ARE: 3 3 2 
. MY'EXPRESSION IS (3+2)»3 . 
WHICH GIVES ME 15 ^ 

THE COMPUTER IS AT 32 — BOJB ISJT 11 , 

IT'S YOUR TURN,, SPINNERS ARE: 3^5 
WHAT ARITHMETIC EXPRESSION- DID YOU FORM? 

•WHAT NUMBER DOES THAT GIVE YOU? 
=> 23\ . . 

Move ranks 7 which is. FAIR 
* The ISSUES are:^ PARENS ORDER STRATEGY ' 

^'•In, this casfe' the tutor picks on the 
' Issue .of parentheses use, ORDER (using 
the spinTter* in a particular order) was 
not an issue, last turn because. none of 
the better moves re'^juired' changing the 
orde^. (In a-- less condensed session, 
the tutor would not comment on two moves 
in a row, ) 

. 1 NOTIC;£ THAT YOU DON'T USE PARENtHESES 
^ERY OFTEN.- I THIN/( YCJU COULD WIN MORE 
OFTEN IF YOU TRIED USING PARENTHESES. FOR 
EXAMPLE, YOU OOU*!) HAVE M*DE THE 
EXPRESSION: (3+'*)*5, WHICH IS ALRIGHT FOR 
AN AMATEUR ! SO^ YOU COULD HAVE BEEN AT ^49 * 
AND I WOUi^ ONLY HAVE BEEN, AT 32. 

WOULD YOU LIKE TO TAKE YOUR TURN OVER? 
=;> YES 

In this case Bob retakes, his torn. ^ The 
players are put in their original 
position and the spinners are the sar.e . 
Retaking a turn^ allows the student to go 
' through ^the . moiions of the suggested 
mov^ for himself and makes it easier to 
remembers 

v bob is at u' — the computer is at 32 
It's ydur turn., spinners are: 3 *5 
what arithmetic fxpre;ssion did you form? ' 

=> l:3+^)*5 

WHAT NUMBER DOES THAT GIVE YOU? 
=> 35 . ^ 

BOB IS AT ^19 THE COMPUTER IS AT 32 

IT'S MY^TURN.. SPINNERS AREi 
MY EXPRESSION IS: 2*(1-2) 
WHICH GIVES me; -2 . ' ' 
X LANDER IN A TOWN.' 

THE-COMPUTEB^^i-S- AT 40 — BOB^IS AH ^49 

*1T'S YOUR TURN.*. SPIN|^ERS /ARE : 1 2 3 
WHAT arithmetic' EXP^fiESSIO/^ DID iOU'F-ORM? 
=> 2+1*3 / ^ - . . • 

WHAT'NUMBER DORS 'THAT G^^VE YOU? 
= > 5 

^ Move rank/ 13 whicki' is POOJ^ 

The issues are: D^?VIDE' MINUS STRATEGY 



Op to this point* the only operations Bob 
has ueed are addition' ajid 
mult ipdicat ion , -so^ division a^jd 
subtraction tecome issues because there 
- are better moyes whith usre them. 

*YOU SHOULDN'T FORG£T THAT IT'S ALRIGHY T0 ■ 

divide, so long as the answer comes out 
evenly/ especially when there is a »G00D 
move not too far away, yo<j should consider 
division as a way to make the right number, 
this expression is a typical e/xample: 
(u2)/3, which would have resulted in a 

TOV/N! THEN YOU WOULD HAVE BE6n Ar 60' WHILE 
i WOULD HAVE ENDED UP AT ^0 . 



BOB IS AT THE COMPUT&R IS AT k6 

IT'S MY TURN.. SPINNERS ARE: 2 k 2 
MY EXPRESSION IS: ^2+(i4/2) 
WHICH GIVES ME ^ 
I TOOK A SHORTCUT 
I BUfiPED YOU 

THE COMPUTER IS AT 54 BOB .IS AT AO 



IT S YOUR TUR'.. SPINNERS ARE: 1 2 2' 
WHAT AJ^ITHMETIC EXPRESSION DID YOU FORM? 
'=> U2»2 

WHAT NUMbER DOES THAT GJVE YOU? 
= > 5 

Move ranks 11 which is POOR 

The ISSUES are: BUMP SHORTCUT ORDER 

STRATEGY 

YOU DON'T SEEM TO BE BUHPTING VERY MUCH. 
BUMPS ARE HARP TO GET BUT ARE USUALLY A 
GOOD ICTEA. ONE GOOD E:?f.MPLE WOULD BE THE 
EXPRESSION: (1*2)+2; WHICH \;OU^LD GI\fE YOU A- 
SHORTCOf AND A BUMP1! SO YOU COULD f/aVE 
BEENi^AT 54 WHILE I WOULD HAVE ENEDED'UP AT 
-40. 

'BOB IS AT 45 — THE COMPOTER IS AT 54 

IT'S MY TURN.. SPINNERS' Ai^E : 12 1 
MY EXPRESSION IS: (!+!)*2 
WHICH GIVES' ME 4 

THE COMPUTER IS AT 58 BOB IS AT^ 45 

IT'S YOUR TURN.. SPINNERS ARE: 2 0 7 
WHAT ARITHMETIC EXPRESSION DID YOU FORM? ' 
=> 2*0+7 ' * » 

WHAT NUMBER DOES THAT, GIVE YOU? 
= > 14 

The WEST system also contain- a simple, 
arithmetic expression o ^ rnostician . 
whlcf) looks Tor mixed up pre ^ jence. 



)N>r5 



MULTIPLICATIONyfS DONE BEFORE 'ADDITION SO 
2»0h-7 ^IS EQUAir TO (2*0 )+V NOX 2»(0+7), 
WOULD YOU LiK'E TO CHANGED YOliR EXf^RESSION? 
=> YES ^ ( 

* 

SPINNERS ARE; 2 0 7 

WHAT ARITHMETIC EXPRESSION DID YOU FORM? 
0+2*7 

WHAT NUMBER DO^. THAT GIVE YOU? 

i> 14 • • 



* Move ranks 3 whS.ch is GOOD 
BOB IS. Ar 59 — THE COMPUTER IS AT 58 

IT'S MY TURN., SPINNERS ARE: 3 1^ 
MY EXPRESSION IS: (1 + 3;*3'* 
WHICH GIVES ME 12 

THE CpMPUTER IS AT 70 1- BOB IS AT 59 
I WIS'. A . ' 

T*HE COMPUTER HAS WON-382, LOST .26 AND TIED 

98 . ' ^ 
BOB HAS WON 0, LOST 1 AND. Tl^D 0 



THAtJKS FOR PLAYING, 



III/ TECHNICAL'DETAIL^ / * • 

. ' The overall opjeratioh of * the 
modelling/tutoring system can be briefly 
restated as follows. Through , the ' course ' of 

,the game',, the student^3>^ behavior^ is 
compared with that of '^'the Expert with 
respect to a' collection of issues . From 
th^e comparisons, a model of the student's 
performance 'ij built b/ the Modeller . When 
the student makes a les's-thah-optimal mfove , 
the Tutor uses the m.odel together with the 
performance of the. Expert to determine an 
issu^ in which the student is weak and 
which would l^a ve ■ result ed - in' a better move.* 
The Tutor then explains the i"5sue to^ the 

.student using ,the better move w*hich 
illustrates it as jgi example. in ^his 
section, we shall desdRbe the issues, ^^he 
Expert, the mq^^l, the Modeller and the 
Tutgr ' which were used to goaerate the 
tutorial . behavior- manifested in the 
protocol. ^ 

The l^ 3 u e s ^ ' • 

The issues define those aspects of the 
environment which are abstracted' into tbe 
model and monitored by the ttj^tori'ng module. 
They provided the organi2in*g concepts which 
coordinate the activities of the modeller 
and the tutor. The issues currently 
addressed are: 



ORDER oi; spinners 



ENtheses 




BACKWARDS 



$pecial moves 



Comment * , 

the spinners don't have' 
lo be , used in any 
particular <5rder. 

the use of parentheses 
Is allowe?[ and is 
frequently valuable, . 

if the result of an 
expression is negative 
the player moves, 

backwards which can 
sometimes .lead vto a 
special move. 

trying for TOWNs, jfUMPs,, 
SHORTCUTS is parf of 'a 



i . MINUS . 

DIVIDE ^ 

■ • "s ■ 

PATTERN 



good strategy. 

subtraction is legal axid 
often useful . - 
V . - . ' 

division its legal ^ and 
sometimes useful. ^ 
/ 

the operations can be 
used' ir> any order, *i.^. 
more than a snail nunbt^r 
of move patterns should 
be used . , 



Modeiler anB the Tutor, 




STRATEGY 



a strategy for Iqoking 
. ' ^ for moves * should be 

*\ ' X ^ ' used','' and alternative 

moves should ' be' 

Considered . 

'Each issue is defined by/thr^e subroutines 
(called prace^Jural s/ecialists ) : (1) a 
Recognizer which determines whether a move 
exhibits the issue; (2) an Evaluator which 
looks at a student model and determines 
whether the student is we5k in tffe issue; 
aiKi (3) a Speaker>» which^ generate: 

. explanatory English ' about the issue.' The 
Recognizers are used by the Modeller to 
update *'^the model r Dn each turn and by, the 
Tutor to ^determi^/ if/ there are better 
moves which the/ sU^dent could have mad# 
which exhibit tl»^^ . irss^ue . The Evaluatbrs 
are used by they^tu'^tor - to evaluate the 
stpdent model iq order to provide E'set of 
possible stufilent weaknesses, ^pefel^rs hre 
used by tMe Tutor to explain the TiiSsue to 
the student, (e^g. f notice t^at-you 

seldom move backwards"). The intent -and 
"operation of each of these specialists w^lj 

^ be described further within the framevork 
of the overall system, - * * 

^ The Expert 
» ' 

Th^ "Expert" module generates and 
^ evaluates the set of moves possible in a 
given situation. For West, the number ^'of 
possible, expressions (varue^)'for each turn 
is small enough that the Expert can 
generate . all of them. Each of^ the 
di^fferent values is then s^^ulated to '.find 
the ^ending positions' of both the player and 
his opponent (renembcr that a player's move 
can "bump" his opponent). in the 
evaluation strategy used by the Expert, trie 
"goodness" of a move is ' the .difference 
bctw<*cn the^ player's final position arlfcj hi^ 
opponent's final position (called the 
"delta"). The Expert determines the list 
of ' legal -moves (ordered from largest to 
smallest delta). When it is the C(^^^JJter/s < 
turn, the Expert need only detei^^e fhe 
optimal move. When it is the stud^^nt '-s 
.turn, , the Expert generates tht^ ent i re - move 
%paco. TYiiT, allows the student's move to 
be Judged relative to the other pos3itjla 
moves that he could have ma^dc.' A5 we shall 
see, the. move*, space is used by both the 



The^ 



.The s'tudeft model is B^^^cord of the 
.student's »ast performance' which provides «' 
the Tutor with information which is useful 
in .determining wat to . say. The model 
consists of a .curtulative struct,ural history 
jof how the.^tudent h^s performed on the 
.•issues relat ivi^ lgith,e performance of the 
: Expert. ' The stt;^uctural model which was' 
bi^ilt by the ^ West system • durfng the 
protocol is ' given in Figure 3 and 
illustrates \ts various^omponents . 



Ifi addition 



to 



the 



cumulative 



structural model , the •system also maintains 
a history ,Jist which • has ^a ^ <^omplete 
'temporal - record of the student's session^ 



:h^^s pinners , 
2 <Pwent , 



iporal 

This includes for each mo^e, 
the expression entered* by t'he UpPUent, the 
results ^of Vhe move .^(bvmps, 'towns,, etc.)^ 
and the final petition. The inf orniaJkion 
provided by the history li^t is needed) for 
example, to check the recent moves made by 
the student . \ ^ - J. 

The Modeller 

The. task of the Modeller is to 
construct and maintain the structural 
model. Using the ^ list of legal moves . 
generated by the Expert^ the Modeller first 
determines an overall "quality" of the . 
student's move. The^quality of the.'move is 
a rough . classification of the ^ move (as 
BEST, GOOD, FAIR or P60R) depending on how 
many better mo^es could h^ve been, mad^e . * 
Each bf 'the issue Recognizers, is then 
invoked to uodate a particular portion ' of 
the model. Bach RecognizerHises thre set of 
better moves to judge the * student's move 
with respect to ita^ particular issue. 

An Issue Recognizer • . ^ - 

Sin^e the major part of the Modelleh's' 
work is done by the individual Recogni^^s 
for each issue, we will desaritie in detail 
/tlie operation of on^ such Recognizer, thd 
^ Pattern Recognizer.** this^ example will 
provide a ^ good overa]/! view of the taslcs 
^ and techJiiques for tjie other Recognizees. * 

The Pattern Recog^ize.r is concerned 
w^th the form .of the expression* i^nderlyinf^ * 
a particular t move . A move is.^class'i f ie3 
into one of 16 pospibla patterns aceorlSlnc 
to the operations u^ed , in thej .expression^ 
and the order in which tWey are per/ormeX' 

*"Bcttcr" is with respect to the Expert 's - 
evaluation proceckjre ( stratcgy-K^hi^h is to. ^ 
maximize) delta. ,5ee below for i^'Xiscussion 
of the possibility of varying strategies. 
**S^e Brown et al [,1975] for a complete 
description of the Recognizers . 



1^ 



PATTERN BEST GOOD FAIR POOR MISSKD/BFST 



Figure 3 Snapshot of a student* modtfl 
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ORDER INFOPJ'ATlOrf: ORIG REV LMS SML OTHER 



GOOD: 
POOR: 



1 

i 4 



ikP^CTION IIIFOm^TIOri: FORWARD BACK\7ARbr 



* GOOD :> 
' ^ POOR: 
WAS/BEST : 



0 
. 0 

0 



iV^T^ENTIIESES : 'NECESSARY 0 OTHER 0 NONE 7 



^SPECIAL M^ES: TOV/N BUMP SHORTCUT 



. TOOK : 
V/AS/BEST : 



0 
3 



STRATEGIES: 

SPECIAL MAXDELTA MAXNUMB ENDGAME OTHER' 



ERLC 



10 



The pattern Bcction profiles the student* s 
use and non-us^. of each ,of the 16 possible 
^mov(i paAtcrns. The ^i*Oh)s indicate tthe numberi 
x>f ti^CB the pattern was used for a move of^ 
each qualitu\ The MISSKD/BEST column indicates 
the number of t;Cmes the pattern occurred a's^OK 
of the optimal mov^s,. *' * 



The totals prouide an overall view of 
strength of the player. 



ihe 



The ranking sectioT^ gives phe dis*trihution 
of how the "Student's moves compared to an' 
exptrt'c. ThcRAiJK of a move indicates how maK 
better moves ther*e were. The t{Ui'WER gives ho'j 
'^many times that ^RAllK yccurved* » ' - 

The order section profiles the order in which 
the spinners were used in the student's move.. 
The ovd^s which are consi'dSred are: ORIG, 
same as vre sented on the spinnei*c ; REV, 
reverse of spinners ; LHS, decreasing order by 
sizoHjr SMLy increasing ' order by size; ari^ * J 
OTHER, nonA of the above. The sub'field^ ' ' 
indicate tne number- of tines the order was 
used when the quality of the move was GOOD 
(BEST or (WOD—i^ pattern section) and POOR 
(FAIR or PSJDR in Pattern section)'. 



The Direction 'section records the number of 
times the studt^nt's expression resulted 
in an initial move FORWARD (and BACKWARD) * 
when the quality of the move was GOO D'*^ (or 
POOR). The was/best field indicates the 
directions of the^ optimal moves . 

The parenthesis section profiles the student 
, use of parentheses }/y noting the numhev of 
flECf.SSARY uses of parcnthes^es as in ^ (A-hB )/C , 
the number of OTHER Mse& ff parentheses and 
t'e number of times no parentheses were used. 

\^ 

The Speciai moves^section maintains for each 
of the sp-ccial^fioveb , 'fOWtls, BU^^Ps, and * 
.SHORTCUTS , how many times ir^he stiXdcnt^ usea 
4hat type of move (TOOK) and tibw rminy times 
the optimal move u^ed it (HAS/BEST) . 

The Strategy section keeps track' of possible 
strategies the student may be using ^ 'The 
strategies arc SPECIAL ^ land me on a special 
move; MAXDELTA , maximize the difference , 
hetiseen your position and your pvponent *s; 
MAXNUMB/ make the ' largest ndimber; and END^GAHE^ 

* land on 70. -iThe counters indicate the nuinber 
o^f student moves which were -optimal under the 
corresponding stratejgy. OTILER keeps account 
the moves which were not optimal unddi^ any of 

r these strategies * , « 



The model contains, for each pattern, t th»c 
numl^er ' of times thC pat^torn was use^, 
subolassif ied by. the qua'lity of - the* move. 
• 'The Pattern section of the model provides a 
profile -of the .student's' use i of. ' each 
6attern and identifies overused patterns 
"Te-g. thos.e which were used 'on f'Aip ,or 
POOR moves).' For exampi.e , it can bp seen 
from, F*igure 3 -tnat' Bob overused the pattern 
A*B+C in the protoc.ol. 

In addition to information about what 
the student did, tne Pattern Recocnizer 

talso mainta,ins a record of what the student 
did not do! 'In particular, for 'those moves 
in which the student 's -move was r^t optimal 
the Pattern Recognizer inCTerr:ents the 
MISSED/BEST field^'for all of the ' patterns*" 

^ Which ^ould have given an optimal mov.e. 
This information points out potential ' weak 
areas by indicating those patterns that the 
student did npt us'e when he should have. 
In general, information , about' what 'the 
student could have done but didn 't is very 
irrportant as it. mi^Bt ' be used to avoid 
criticizing t..e student about issues, which, 
.were never to his advantage to -use. 



The Tutor 

• ,The Tutor i^ responsible for deciding 
what'to say and when la say it. V/ithin the 
"issue and examples" paradigm, the range of^ 
possible "whats" is determined bv^ thc# 
issues that are defined by. the -authofrs of 
the system. Exa9tly which i&suc and when 
,it should be rentioned are ^determined by 
the " student 's behavior and ' a particular 
tutoring strategy. When thje student makon 
a ' less than optir^al move, the Tutor 
recognizes t^e event a? an opportune tey to 
gcnerate%ad vice . The Tutor caris the issue 
'^valuators, (de'ccribed^in the next section) 
to deterxiine * the .issue^f on which . tho 
•student is weak. The list of weak issues 
canslitutes the things the Tutor v/ould lik^ 
.to .tell th6 student about. ' However, just 
because a student 'is weak in somethin:: 
doesn't mean that^ this i.=? the time to tell 
him about -it. * The student will only b3 
interested if >bj/* usi-ng the issue he couln 
have done better. Henc^ the Tutor ^ses the 
list of moves generated by -rthe Expert, 
together ^with the I-ssue "RecQgn i2^ers , to 
determine if any N^f the better moves 

in which th*^ 'student i-i 
i.o.T^es which result fron 



involve an issue 
wcaft. The list of 
this process . can 
Tutor's hypotrtesee 
didn't make 
the student 



be thought- 'of as thf 
about why the st.udcnt 
a bet t cr rr.o.ve . IL ?or examole / i T 
has never used parentheses, and 



the best/ mo v^ requires parentheses, one 
S4jch hypothesis* is that he doc5n"t ucc 
-parentheses. If the list of "tutorable" 
issues is empty, >the tutor has nothing 
particular to say. It pa.n make a general 
comment *"anyway ("I tiiink I <3ee a move which' 



would get * ^ou farther*..") but usually 
would**not do so. If the list tiafe more than 
one element, a' choice betwee»n tlji^ issues 
must * be made. At present,- an* ordered 
"issues list" rs Viaintained which gives the 
relative im|pBrtance of each issue. In a 
more complex domain; the Issues could tiavtf, 
for example, a lattice structure "where 
r cert^n issufes are prerequisite to other 
issues J» * ' ' ♦ » 

Once the Tutor^ has chosen ah issue and 
an example,* the Speaker assoQaiate<l with 
the issue is invoked to provide feedback to 
the student. At present ttie Spe^W^rs are, 
very, .simple. Each has thr^ee or four 
possible phrases for each of t|irere' or four 
parts of an explanatory paragraph. This 
implementation h^s the advaotag^s of being 
easy to build and providing a Veasonable 
variety of, comments. The main I'imitation 
of such i^implicity is that a Speal«^r whicl) 
is ,not aware ' of Hhe <?orjtext in„ wijich it 

jnust "talk" (i.e. player prosit ions , *'?ioves , 
etc.) mu^ make very general commenujs (or 

''risk making * inc^pprppr iate comments)*-, and 
hence miss chance^ for bei*n^ part icul^riy 
incisive. - ^ ^ 

An I^ue Elvaluator 

The success of the Tutor \depends 
critically on the ability of. the Issue 
'Eialuators to isolate the^ weaknesses\ Of the 
student. As an ♦•e>;ample of t'h^ t^pe of 
operatioji performed b/ the« Issue 
Evaluators, we will ^describe the Pa\tte.*v:'^ 
Evalnator. The Patt*n Evaluator cfi^ecks 
the student modei. (see Figure SJ'-to se^ if 
the student Is varying the form of \his 
move. ' The important factor is how the 
student 's behavior compares with t^he 
^Expert's. That is, how oany times has the 
student used a^ given pattern when' he .could 
have done batter v/,ith a different one. As 
mentioned earlier, the Pattern Recogniz^er 
classifjj&s each move a$ one of l6 patterns 
depending on the operations and their order 
of ''operation. Thus if th^* ^tud^ent .'is 
always forming A+B*C, that/field of the 
pattern seq,tion* will have a large portion 
of the moves. *Notice, however, that 
constant use. of a single pattern does not 
necessarily indicate, that the stuc^ent is 
stGck. It may ^be t,he ca* that in the^e 
articular situation.*?, the student made the 
iTGSt move, ^'or this reason, t^e ^valuator 



1^ 



* •The best such hypothesis is dne which 
^ exhibited by al I of the better moves. * 



i"s 



*The tutor also uses ' other strategies to 
limit its verbosity such as not hassling a 
studoint ^fl' an^^issue ' he has pehform.ed 
satisfactorily within the last three moves,^ 
not hassling the 'student two moves^ in a 
row, *hot hassling a poor player on' a GOOD 
as opposed to BEST move; ' This type of 
tuning is critical a smoothly operating 
systemi.but is; at ^r<^scnt, vary^ad^hoc an'd^ 
^will Ijot be mentioned further' in this 
paper. 



ERLC 



Lool^s- at* the Jion-optlmal subflelda of 'each 
pattern to deterniine how oft^n the stSdent 
usjed a form when it was not,' the optimal'/ 
;thing' 'to do. s The criteria the Pattern 
'Evaluator usefel to determine" if thC^ , student 
is Styck in. a pattern' is: "Has 'the student 
used this pattern /lon-optiniaj.^* V*'"o.'*6 than 
75% ' of tKe; tines, that h^* ba5,aot used i't 
optimally. *^ ' . ' " * ' ' 



IV. 



DISCUSSIpM ;> ^ * • ' 

When we- begah,sdes,iJaniTi^.thifc,^y3tem 
faced uncertainty abo'i^i^Jl^-hailt^s*)^ go ihto ^ 

^a Student modoi^ahd how^lo^^^^i(Tcr the tulor 
^nto^ makings irfsightful *"*conments at 
releva^nt, * and .o-n^y relevant, times. 
Because of the lapk ;'of any conprehensi 
theory for how to grow gnd use studejpt^ 
models or what constitutes useful tiitori^al 
co^iments, our system was •desipned ^o tl^t 
it could be easily modified. ' That Hay,'*''we 
could run subjects on thiQi^ systemj observe 

^the syst'jem 's behavior and* the students' 
reactions, modify .the' system vVhere 
hecessai'y ^^nd *eventj^l'ly compare the 
system's behr.vio'r to that of human tutors 

• ( ourselves ) . ' • ' , ' 



% •'We , would * like ' to d^fccijibe ^ two 

techniques We* found useful fc^r maluat ing 
successive versions 'of the VJestf system. 
Ofie ^^"iSe th^d , whi^ v/e used to deterninc tf^ff 
adequacy of the mo^ly was yto see if a 
human tutor, using just the , nod^l 
constructed by^the Modeller and a given, 
student's ♦move, could determl>s^^^^^ the 
studtint 's weakness. ^When the r?rt^dcl 
contained so little 6r poorly structured 
information, that a persoi# could not 



generate reasonable 
reason to belip-ve, 
neciessarily do afp , 



comments, ^we 
a program 



saw no 
should 



The other ' method 



our 



playing V/est y under 



involved 

consist en t but suboptimal strate'gy such as 
always using tjiG-spinn^s in theVam^ ordef- 
, and never using parentiheseVT ' fin . sucn 
■ circumstances th<^ tutor should coniment oxi 
those (and only those) issues- whichl we^ were 
•purpose full^yavoidi ng . ' 

Eyperinen tal Rc.r.ul I s 



A] though 
hiajor i^tudies 
^sys'tem is, we 
• fexpc ri men t s 



ru..nlng 1^8»^tHdent teache^rs 
one* used the syster. for at 
A f t e pv/a r;i G , * ea c h \.-a s a c. k e d 



wo have not'^yct* (inducted any 
of how effe<itJ:ve our tutorinpr 
have ' run - several informal 
One of ^ thos^ cons'i^fCpd of 
in which each 
least^^e hour^. 
to conplete a 



quostaonr^air^ ,. v/lth ^ 12 <:;ornply ing . ^Tho 
following* conncnts apply t^o' this NS.ampl«: * 

All bOit one subject received adii^ice 
from t|?!e 'Tutor . ''•"Thei r. consents abouf the 
Tutor w^re quite favomb-ie. Nincf subjects 
stc'ted that the T'utor's cornonts , were 
appropriate to what they were doing. Of 
the two who disaigreed, one said that the 



Tutor was' offering, a ^rategy vwich he 
didn/.t -feel he 'should /'oLlow becUu^e it 
vroulij leave him "yjhne-r^le to attack**,^ an 
elemei;it of strategV not known* to 'our 
.dTurrent Exper^t^. Ei^ht out oC^'ten subjects 
foun^d the -comments 'helpful in learning a 
better *way to . ^plaj^ tK^ , game an<l most 
important,' nine/ou£^ of ten felt that the 
Tutor manifested a " ^ood * understanding of 
their weaknes^ies ! One' su^bject commented-"I 
misunderstood a rule;* the computer picked* 
it up in the 2nd game." • * - , ' 

' ' * ' \ 

We 'are , quite •encouraged,'f*by these 
resululs. Not only^ did the subjects^ sense 
the "intelligence" of -the ' Tutor**'in Knowing 
when to offer appropriate .suggest ions bat 
they seemed ,\,6 enjoy the Tutor '^ , support. 
W-e^ of course, ' realize that thi,s data -is 
'^"^/ghly subjective and are 
• to , cgnducting. some 
experiments 



looking forward 
more "controlled 



Ex teas ion a , ' 

While the present ^systenj- has Vorkei 
very well in experiments, there are seVei^al. 
extensions - / to ^ the paradigm / woF.th . 
mentioning. One deals with tl^e problem of 
^'changing the-point'of view", of the student 
model. T)he'" system* evaluate's a move ba'^ed 
on'^its comparison to an E:^pej^b's. move in* 
th^ same ^ituat^-on. This Expert must use * 
s^oma- st rategy to decide whicft move is best. 
Fqr exampl^^^ is '^it ' better to 'get 'one 
farther or to - t5e on a -t own? ' V/l»et ever 

s strategy Che Expert uses (rt currently uses 
the maximum delta , strategy ) ,^ it may not be-^ 
the same strategy employed 'by the student. 

*When*this is the case, the stucfent's ^ovfes • 
woa't be evaluated^ corredjt'ly /using the • 
Expert"s straterv as a §ta»ndardj^ Ilj^the^ 
reason for tutoring the stu^Tent not 
n$cessarij.y to teach *hirn the' -^^xperf's 
notion of a goocl strategy, but instead to 
help him become .aware of a wide range of 
issues, it mrght be beneficial to c'riticize 
the stud"ent within his- own strat^egy^ If we* 
discover: that the student is playing^a 
coherent but different strategy (either by^ 
as^king ^hi^- ^r b'y noticing pat-fcerns in his - 
Uodel**) the Modeller can re-synthesize^ the 
model using the history ligt- and an Expe*rt 
who ^imuTates the pa reticular student 
strategy. When the Expert' correctl-/ 

'guesses and * simulates the' actual " strategy, 
the resulting mo*del wili sh^rpl^^ 'indicate a 
•better player. *At thi^ point, if . we 
verbalize .this_ 'strategy to the- studen-t f we 
can mak'e him/awar^ of it and -hetice ' willing' 

..to consider al t erna t i veaf'. This provides 
hifn with a goal* in additipn to Vrithmetic 
practice, 'i.e. .he can exper*lment with 
strategics^.' ' , ' 



*The types of^patterns in the model' might- 
be a large, number of moves which are notf 
.optimal' in any knc/wn- strat^egy together with 
general strengths J^n, other .areasj i.e# 
when the student , is' making less- tha^ 
Optimal moves which can't tJe ^explained by 
ihe issues. ' ♦ 
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* A rtoVe ^enqral \imitation stems '.ifrc^ 
the • issues^ and .examples paradigm U^elf,. 
At? present the l*3sues *act 'lik,e "demons'! 
observing ' the student's 'activities/ 
watching for sit uations..in which they'»>Span 
' point -'out something of interest.- *This 
technique is very gpod at takmg* advantage 
of ^ the work ,*that the student', has* done to 
poi^t out *i«teriesting things. there .is 
another 'dj.mehsion«^, to , tutoring which this' 
tcchnij|i>e does not capture. - That is the^^ 
not ion\of ^.dr^ac£ing the studjpnt activit^io^ 
\in a general' direction in * the hope of 
/putting him in tan in^terestin'g situation.* 
For' example,* ' if a- particular, .issue has 
--ne^^er .come up, we could bias the ^spinner 
values to try Jto make that issue * arise. 
V/hile^ our cur;rent systera doncentrates on . 
'^bo<tom-up tutoring ^through-' issues 'itr . is 
'^iear* to us that a general system must also 
inplude tojj dowrt guidance. .f 



.ihis system was fuijded iri'\ct*tt .mde?* a 
fr.i-Service' '(AFHRL, ARIf,- ' . ARPA 

^HRRO) - contract.. * Th% ^/^^.^^^h * ^ wa^ 
conducted as palrt of a ^jt^^^ ' .iiscover 
better ways to con6tr^ct 'diy\' ju^^s r^uctural ^ 
models of the learner for ira^"ing 
^ appl4cations ^uch »a^s aur^." ' tt-^ training 
• simulators with int'elligenc 6>i4>puXier-i based 



onclusions ' * • 

* * 

The overall sense we had Trom building 
and experimenting with this system is that 
It is easy to talk, about student mod'els and* 
yet . surprisingly difficult tp act'ually 
construct a%^ system that can grow an' 
in^sightfuli rn^e'l of the student ^apd then* 
use this Nfedel in a sensitive way to Itut.or 
the student. ^ The. [pedagogical ' value of 
flrawing tutorial exanpl^es - f^rom 
student work seen's beyond rej^road^-, 
the int^ll-igenoe ,the system nusl have 
successfully. act * ' on I ' 
considerable. 1 GoQst ruqt ing 
cdn^tar^tly cri-t^iciSels 
' straight forward Jhe point i'S vto make ooe 
that' only . interryijits when a sk:^l led ^ hunan 
tutor-would and then generates 
remedial coinmervt ; . ' ' 



the 
yet 
to 

its own is 
a tuto^ . which , 
is relative!} 



succi'not 



tfe feel tha^t.out West system .arxJ, . the 
genqraJ tutori'n'g pjiradigm of "Iss1!^es" and 
'ExampleV* provides the beginning . pf a 
theory of how^thi's can be accoapllshed J ' It 
also pi^ovides .a ■ glimpsa .of tiie , techniCcjl 
problems wfiljt?h* must' be 6otif*ronted in 
actually construe^t ing* an operational system 
that can -grow *anS use ^ s.tud.ont' models in' a 
versatile way. ...'^ t , ' ' 



tutors , 
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